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What the heck aoes that mean? 







High-performance computing often 
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specialized hardware 



p 



1 ♦ Supercomputers 





♦ experiments with graphics p 

General-purpose computing doesn^t 
optimize for technical computing 
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With 



some 




ro 




ems 



Sup 
Exp 



ercom 



puters 



ensive 



Not on the same 
teen 



noloeu curve 



w 



Different programming 



environment 



General-purpose 



com 



puting 



Amazing technology 



curve 



Optimized for desktop 
and enterprise 
applications 
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alienee: 



ACh 
Trie Best of Both 
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Use general-purpose hardware components 
With a standard programming environment 
And SYSTEM DESIGN for technica 



put; 



comoutin 
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But all is not well in 



supercomputer Ian 




cL. 






You have to pau a lot for them 

You have write uour program dirrerentlu 

You have to find some high priests to 



talc 



e care o 



fth 



em 




Supercomputer companies don^t make 



mone 
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et's use lots or littl 



e 



com 



puter: 



5 






PCs are cheap 

Linux is free 

Commodity interconnect (Ethernet) is 



chea 



p 

Tne (Beowulf) Cluster is born 
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tech 



nical computin 
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Some typical applications 



Climate and weather 



mo 



del 



Geophysi 



sics 



Complex financia 



mo 



deli 



in 




Mechanical design 



Pinite element analysis 
rluid dynamics 



Lif 



e sciences anaiusis 



A 



3 



ana simu 



ation 



Top-secret stuff 



...and 



manu otners 



y 



th 
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What 





e 



Can run for weeks 



Consume all the cycles 



3 



ou can arron 



fforcl 



Not very cache-friend It) 
Parallelism often 



d 



d 



emanas eoo 



d 



communications 





e 



*? 



Large data sets (input 
and output) 

Many are in Portran! 

...but also in Q C++; 
Java^ Perl j Python, etc. 
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Trie Market for HPTC 
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HPTC is now mainstream computing! 





put; 
Over $6 billion in Linux cluster hardware 



1 sales in 2006 




Petascale computing is hot for research., 
but there is a real market now for 
teraflops 
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Cheap, b 



ecause 



th 



: y 



use chea 



p 




Exp 



an 



dabl 



Easy to get started 



Softw 



are 



isf 



ree 



Th 



eu ride the desktop/ 
server tech nolo£u curve 



w 



great 



) 



n 




ht? 



Interconnect (Ethernet) 



i 



is chea 



p 



Emerging de facto 
standards 



Linux 



Message Passing 
Interface (MP!) 



Q Portran, etc. 



_-* 



— *■* 



■»«t-i 



— 



■^■■PW 












1 
i 



^i 



V 






1 









J 



■ 



...but not perfect 



utationa 



Comp 

efficiency is often low 

Use lots of power 

Generate ots of neat 



Many parts to fail 



...with a desktop MT5P 



d 



esi 



S n 



Interconnect is slow: 
XXX microseconds for 
MP! on Ethernet 




...or expensive: usin 
Infiniband can increase 
the price of a node by 

50% 
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And 




ftw 



are rules! 




Software investment is the significant 



i 



cost 




Replace the cluster, but keep the 




ftw 




What if we redesign the system 



W! 



thth 



e same programmm 
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A Design Challenge 



1000 nodes in this box 



...all running Linux 

Near-microsecond MP! 
latency 

Air-cooled 
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The logic or low power 








Low power 



Less heat 





heat 




parts closer together 



th 






Parts closer together => shorter wires 

easier high-performance interconnect 
Less heat => greater reliability 

Burn less power waiting for memory 
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tK 



e 



55^2 Gigaflops 



777^ Gigabytes ECC memory 

5^72 <?~core ^f -bit nodes 

1^)6 2 GBute/s fabric links 

about 1 microsecond MP! 
latency 

105 5-lane PCI-Express 



\S KW 



1 Cabinet 
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tK 



e 



SC6+8 



6^rS Gigaflops 
S6& Gigabytes 



RAM 




103 6-corc 6^r-b\t nodes 



'!• CniSe* 



324 2 GB/s fabric links 

about 1 microsecond MP! latency 

12 3-lane PCJ-Express 




SCI 




2KW 



1/2 standard 1^ M rack 
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Softw 





It^s iust Linux 
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MPI 






...even Emacs! 




All 



open source 
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Interconnect fabric 




Log diameter 
Multiple paths 
Cost-effective 
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A Cluster Node Chip 
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CPU 



CPU 



CPU 



CPU 



CPU 









LZ Coherence Engine 




DMA 
Engine 





Memory 
controller 



Pabri 
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Memory 
controller 
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Exp 
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switch 



RAM 



RAM 
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Design for reliability 
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Lower parts count 



parts 



♦ Lower power ~ less heat 








All RAMs have 
Redundancy in interconnect 






■»«(.■ 



- 



■^■■PW 












1 



^1 



■ 






1 









< 



■ 



Paralle 




I/O 




Integrated Lustre cluster filesustem 
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Op 



en source 




POSlX-comoliant 



p 




Multiple uses 






Direct-con nect storage 
External Lustre servers 
RAM-oased filesustem 
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What have we learned? 



■ 




Take genera! computing techniques 





...with some knowledge about the 
applications 

Mix well 

Powerful and usable computing 
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